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Abstract 

This paper is concerned with a trust-region ap- 
proximation management framework (AMF) for solv- 
ing the nonlinear programming problem, in general, 
and multidisciplinary optimization problems, in par- 
ticular. The intent of the AMF methodology is to 
facilitate the solution of optimization problems with 
high-fidelity models. While such models are designed 
to approximate the physical phenomena they describe 
to a high degree of accuracy, their use in a repetitive 
procedure, for example, iterations of an optimization 
or a search algorithm, make such use prohibitively 
expensive. An improvement in design with lower- 
fidelity, cheaper models, however, does not guar- 
antee a corresponding improvement for the higher- 
fidelity problem. The AMF methodology proposed 
here is based on a class of multilevel methods for con- 
strained optimization and is designed to manage the 
use of variable-fidelity approximations or models in 
a systematic way that assures convergence to critical 
points of the original, high-fidelity problem. 

Key Words: Approximation concepts, model man- 
agement, surrogate optimization, nonlinear pro- 
gramming, multidisciplinary optimization, multilevel 
methods, computational engineering. 

Introduction 

This paper concerns a globally convergent 
approach — a trust-region approximation manage- 
ment framework (AMF) — for controlling the use of 
approximations or models of variable fidelity in solv- 
ing the nonlinear programming problem (NLP): 

minimize f(x) 

subject to h(x) = 0 (1) 

g(x) > 0, 

where the objective / : Tt” — y and the equality con- 
straints h : W 1 — y and the inequality constraints 

g : t are sufficiently smooth nonlinear func- 


tions with m,n,p > 0 and m < n. 

Mathematical models of physical phenomena nec- 
essarily provide only an approximation to the “true” 
entities they describe, that is, all computational mod- 
els are approximations or surrogates of the underlying 
function. In some publications, special meaning is re- 
served for the terms “approximation” , “model” , and 
“surrogate” , and the meaning differs from author to 
author. In the context of this paper, the terms are 
interchangeable . 

High-fidelity models, such as the Navier-Stokes 
equations of aerodynamics, are designed to approx- 
imate physical phenomena to a high degree of ac- 
curacy. However, their use in repetitive procedures, 
for instance, in iterations of an optimization algo- 
rithm, can be prohibitively expensive. On the other 
hand, an improvement in design with lower-fidelity, 
cheaper models (e.g., the Euler equations for aerody- 
namics) does not guarantee a corresponding improve- 
ment for the higher-fidelity problem. A natural ap- 
proach to alleviating this difficulty is to alternate the 
use of higher-fidelity approximations with the use of 
lower-fidelity approximations in a single optimization 
procedure, based on some rules for deciding when to 
switch to a model of a different fidelity. 

Approximations have been used in engineering 
optimization for a long time in various procedures 
based on heuristics (e.g., [1, 2, 3]). A survey on the 
use of approximations in structural optimization can 
be found in [4]. Reports of recent efforts in devel- 
oping methodologies for “variable-complexity model- 
ing” can be found in [5, 6]. With a few exceptions ([7], 
[8]), until recently, the analysis had focused on the 
question of convergence to a solution of the surrogate 
problem ([9], [10]). The intent of the AMF methodol- 
ogy proposed here is to facilitate the solution of opti- 
mization problems with high-fidelity models by alter- 
nating their use with the use of lower-fidelity models 
in a systematic way, resulting in a procedure that is 
globally convergent to a critical point of the high- 
fidelity problem. 
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Work on approximation management frameworks 
may be roughly categorized into zero-order [11] or 
derivative-free [12] methods and first-order methods 
[13]. Formally, the zero-order methods are those 
based on direct-search algorithms that do not build 
models of functions and, hence, require no derivative 
information from the user, while first-order methods 
explicitly rely on derivative-based models. In prac- 
tice, however, the distinction is not as pronounced, 
because zero-order frameworks are hybrid methods 
that use a direct-search component but do build mod- 
els with approximate derivative information. Because 
first-order frameworks rely on derivative-based mod- 
els explicitly, however, to date only the first-order 
frameworks have been extended to handling general 
nonlinear constraints. The question of when it may 
be preferable to use a framework of a particular order 
is open and is a subject of ongoing research. 

A provably convergent first-order AMF for uncon- 
strained optimization was introduced in [13, 14, 15]. 
It was based on the trust-region methodology (e.g., 
[16, 17]), which can be described as an adaptive 
move limit strategy for improving the global behav- 
ior of local model-based algorithms. The work in [13] 
dealt with unconstrained optimization for two rea- 
sons. First, many algorithms for solving constrained 
problems are reduced to solving a sequence of un- 
constrained subproblems. Thus the analysis of un- 
constrained problems provides a foundation for the 
treatment of constrained ones. Second, the introduc- 
tion of the idea was eased by addressing only uncon- 
strained problems. However, the need to solve con- 
strained problems was obvious, and in [13], we stated 
that the extension of the unconstrained AMF to con- 
straints via, say, the augmented Lagrangian approach 
[18] was immediate. This extension is done in [19]. 
Another extension of the AMF in [13] to constrained 
optimization has been proposed in recent work [20] 
that uses multiplier methods together with response 
surfaces and the concurrent subspace optimization 
method [21] for multidisciplinary design optimization 
(MDO). Convergence of the latter method has been 
conjectured. 

The AMF methodology discussed in the present 
paper concerns a scheme based on a class of multilevel 
methods for large-scale constrained trust-region op- 
timization (MAESTRO) [22, 23, 24, 25]. The result- 
ing AMF manages the use of variable-fidelity models 
both for the constraints and the objective in a sys- 
tematic fashion that preserves the global convergence 
properties of the underlying class of algorithms. 

The idea is as follows. Given high-fidelity mod- 
els f(x), h(x), g(x) of the objective, equality, and 
inequality constraints, respectively, of a physical pro- 


cess, and a suite of corresponding lower fidelity mod- 
els { a{(x)}, {a^ 1 (a?) } , {a?(*)}, the overall “external” 
framework is that of a suitably modified foundation 
algorithm, in this case, the MAESTRO class of algo- 
rithms. The computation of the trial steps in the 
external framework is then itself a set of iterative 
procedures. These procedures bear the brunt of the 
computational expense and they are done by using 
approximations to the lower-fidelity models. The al- 
gorithm resorts to high-fidelity computations only pe- 
riodically to evaluate progress towards a critical point 
(or a solution) of the high-fidelity problem. 

The multilevel AMF is capable of handling arbi- 
trary models, provided that a set of consistency con- 
ditions defined later in the paper is satisfied at some 
points of the iterative procedure. For example, AMF 
is not limited to the use of algebraic, Taylor series, 
or response surface approximations. Analyses of vari- 
ous physical fidelity, such as Navier-Stokes and Euler 
codes, can be used as variable-fidelity function evalu- 
ators in AMF. In current demonstrations, we are par- 
ticularly interested in managing aerodynamic models 
of varying physical fidelity. 

The paper is organized as follows. Section 2 gives 
a brief overview of the underlying class of multilevel 
algorithms. Section 3 introduces the bilevel AMF in- 
tended for solving conventional NLP with a single 
block of constraints. Section 4 describes a multilevel 
AMF for large NLP or for MDO problems. Section 
5 contains some very preliminary numerical results. 
Section 6 concludes with a summary and some re- 
marks concerning ongoing research. 

The notation will be introduced as necessary. All 
norms in the remainder of the paper are £2 norms. 

Background: 

MAESTRO class for nonlinear optimization 

This section contains a brief review of the under- 
lying MAESTRO class of algorithms for constrained 
optimization [22, 23, 24, 25, 26] without the inclusion 
of the approximation management. 

The current AMF demonstrations at the Mul- 
tidisciplinary Optimization Branch (MDOB) deal 
with relatively small problems — 0(10) variables and 
constraints — with expensive function evaluations. 
The general inequalities are managed via squared 
slacks and, therefore, the remainder of the paper is 
concerned with the nonlinear equality constrained op- 
timization problem (EQC): 

minimize f(x) , , 

subject to h(x) =0. ' 

The MAESTRO class of methods for constrained 
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optimization is a class of algorithms motivated by 
MDO problems with arbitrary couplings but applica- 
ble to large-scale optimization problems in general. 

In this approach, the constraints or equations of 
the system are partitioned into M blocks 


h(x) 


/ hi(x) 

\ h M {x) 


where k = 1, . . . , M is the number of the currently 
processed constraint block and the number of the 
subproblem. (Note that for k = 1 there is no null- 
space constraint.) The points y k are then set to 
Vk + s k . Once the constraints have been processed, 
the substep sm+i on the model of the objective func- 
tion is computed by approximately minimizing the 
quadratic model <f>(s) of the objective in its own trust 
region, as follows: 


in a manner dictated by the application. In the con- 
text of MDO, disciplinary boundaries provide a nat- 
ural partitioning into blocks. No special structure is 
imposed on the problem. In particular, narrow band- 
width of coupling is not assumed, and the subsystems 
of the problem can be, in principle, fully coupled. 

Once the system is partitioned into subsystems, 
given a current approximation to a solution, the 
next point is computed in a sequence of progressively 
lower-dimensional trust-region subproblems, each of 
which computes substeps on its own block of equa- 
tions or constraints, subject to maintaining the pre- 
dicted improvement already obtained for the previ- 
ous blocks. The final substep is computed by mini- 
mizing the model of the objective function or of the 
Lagrangian, subject to maintaining the predicted im- 
provement in the models of all constraint blocks. The 
total step is the sum of the substeps computed in the 
subproblems. 

In particular, suppose x c is the current ap- 
proximation to a solution of problem EQC. Let 
Si, . . . , Sm+i be the trust region radii for the M sub- 
problems that deal with constraints plus the sub- 
problem that deals with the objective function. The 
trust-region radii bound the regions in which particu- 
lar models are “trusted” to approximate the behavior 
of the corresponding functions. (After the trial step 
is computed, the actual behavior of the merit func- 
tion is compared to its predicted behavior and the 
trust-region radii are updated in a systematic man- 
ner.) Further, let t/o = x c . Then the trial step s c 
is computed as a sum of the substeps s k , each of 
which is an approximate solution of one of the M + 1 
subproblems. During the multilevel constraint elim- 
ination procedure, each substep s k is computed by 
approximately minimizing the Gauss-Newton model 
of a particular constraint block, subject to the null 
space constraints that assure the maintenance of the 
predicted improvement already obtained for the pre- 
vious blocks of constraints. Specifically, the following 
subproblem is solved: 

minimize ± || h k (y k - i) + V/i^(t/ fe _i)s || 2 

subject to V/iJ (t/j_i)s = 0 , j = 1, . . . , k — 1, 

|| s || < S k , 


minimize <j>(s) = /(j/m) + V/ T (t/ M )s + \s T B M s 
subject to V/iJ (t/j_i)s = 0, j = 1, . . . , M 

II s || < $M + 1, 

where Bm is the Hessian of / at j/m or an approxima- 
tion to it. The only assumption placed on the approx- 
imation of the Hessian is that its norm is uniformly 
bounded from above. In particular, the Hessian can 
be zero, resulting in a linear model of the objective. 

The total trial step s c from the point x c is the 
sum s c — Si T . . . T s M+ i . 

The salient feature of the trial step computation is 
that each substep s k is assumed to solve its subprob- 
lem only approximately. “Approximately” means 
that each s k can be obtained in any manner suitable 
to the application, as long as it satisfies two mild 
conditions. Specific ways of computing the substeps 
will give rise to different members of the MAESTRO 
class. 

The first of the two conditions on the substeps is 
a sufficient decrease condition known as the Fraction 
of Cauchy Decrease condition (FCD). It concerns the 
change in the model of the objective function or con- 
straints from the point x c to the point x c + s c for 
the entire trial step and from points y k -i to points 
y k -i+s k in computing the substeps. Roughly speak- 
ing, FCD says that a particular model is required to 
predict a fraction of the improvement in the objec- 
tive or constraint functions that is predicted by the 
steepest descent step, i.e. , the minimizer of the linear 
model of a particular function, restricted to the trust 
region. Let h k denote half the gradient of the norm 
squared of the constraints, i.e., 

hk = Vh k (y k _i)h k (y k _i). 


For each constraint block, the FCD condition can be 
stated in the following form: there exist cr > 0 and 
T > 0, independent of the iterates, for which the sub- 
step s k satisfies 


h k (y k - 1 ) || - || hk{yk-i) + Vh k (y k _i) T s k || > 

h k 


h k 


S k 
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For the objective, the FCD condition implies that the 
substep sm + i satisfies 

/(2/m) - + sm+ 1 ) > 

llnt , M, • ( S l|V/(j/M)lh 

o' II V/(t/ M ) || mm U M +i, j; I • 

The constants cr and T need not be same for the ob- 
jective and the constraint blocks, in general, but they 
can be made the same. 

The FCD condition is easily satisfied in prac- 
tice. Under the assumption of FCD, strong conver- 
gence statements can be made about trust-region al- 
gorithms for unconstrained optimization [17, 27]. For 
the MAESTRO class for constrained optimization, 
an additional boundedness condition is necessary. In 
particular, each substep s k , k = 1, . . . , M is required 
to satisfy 

|| s k \\<IC || h k (y k _ i) || 

for some constant K, independent of the iterates. 
Again, K, need not be the same for all blocks of con- 
straints. This condition is also very mild. It assures 
the existence of pseudoinverses for the derivative ma- 
trices of the constraint blocks, and it allows for a wide 
variety of step choices. 

The trial step is evaluated by comparing the ac- 
tual improvement in the merit function with the pre- 
dicted improvement in the merit function. The step 
is accepted or rejected based on the comparison, and 
the trust-region radii are adjusted. The merit func- 
tion and the step acceptance and update procedures 
are described in more detail in the next section. 

Given the two conditions on the substeps s k , un- 
der a number of conventional assumptions of the non- 
linear programming analysis, the algorithms of the 
MAESTRO class have been proven to converge to 
critical points of problem EQC [22, 28]. 

The mildness of the requirements placed on the 
substeps assures that the class is very extensive. This 
attribute is significant from the application perspec- 
tive, because in MDO, the blocks originate from dif- 
ferent disciplines and almost certainly require differ- 
ent solution methods. Since each substep is computed 
autonomously, it has been noted ([22]) that MAE- 
STRO would provide a natural framework for deal- 
ing with different models for MDO in a systematic 
fashion. 

Bilevel AMF 

We first consider the case of a single block of con- 
straints, or the conventional NLP, in which the con- 
straints are not partitioned. The trial step in the un- 
derlying algorithm is composed of two substeps, first 


on the constraints and then on the objective, result- 
ing in a bilevel MAESTRO-based AMF procedure. 

In the MAESTRO class, as in classical trust- 
region algorithms, the amount of optimization done 
at each trial step is controlled by varying the size of 
the trust region, because only one model is used for 
constraints and one model is used for the objective 
during all optimization iterations. Because the mul- 
tilevel AMF inherits the global convergence proper- 
ties of the original class, it is not necessary to change 
models to obtain convergence. However, in the case 
when other models are available, such as varieties of 
structural or aerodynamic models, MAESTRO-based 
AMF provides the user with guidance on changing the 
nature of the models. 

Computing the trial step 

In the bilevel AMF, to compute the trial step from 
the current point x c , we first select a model of the 
constraints that satisfies the following consistency 
conditions for the constraints at that point: 

a h c (x c ) = h(x c ) (3) 

Va h c (x c ) = Vh(x c ). 

We find a substep si that approximately minimizes 
that model within its trust region. The process of 
computing the substep is itself an iterative procedure. 
Next, we select a model a l of the objective function 
or the Lagrangian that satisfies the consistency con- 
ditions at the just computed point: 

a { ( X C + s l) = f(x c + Sl) (4) 

Va{(x c + «i) = V/K + Si). 

The substep s 2 is computed in another iterative pro- 
cedure to approximately minimize the model in its 
own trust region. The total trial step s c is the sum 
of the two substeps. 

Consistency conditions (3) and (4) are the condi- 
tions used in the unconstrained AMF [13]. The con- 
ditions can be relaxed, but we will not pursue that 
line of reasoning here. Even in the form of (3) and 
(4), the conditions are not restrictive, as they can be 
easily enforced [29], given any two models. 

Consistency requirements are placed on the mod- 
els used at each optimization iteration of the external 
framework. No such assumptions are made during 
the inner iterative procedures when the approxima- 
tions of the lower-fidelity models are minimized re- 
peatedly. 

Numerically, let A c £ 3? be the current value of 
the trust-region radius. Then the current trial step 
s c will be computed as follows: 
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Set z o = x c 

Select a*, such that a(?(z o) = h(zo) and 

Vaf(zo) = Vh(z 0 ). 

Compute si that approximately solves: 
minimize a(?(z o + s) 
subject to ||s|| < 9A C , 6 £ (0.5, 0.6). 

Set z i = zo + s\. 

Select al , such that a^(z i) = f(z i) and 

Vaf(z 1 ) = Vf(z 1 ). 

Compute S 2 that approximately solves: 
minimize a l (zi + s) 
subject toVa(?(zo) T s = 0 

Ml < — IIMI 2 - 

Set s c = si + s 2 , 

where 9 £ (0.5, 0.6) is a somewhat arbitrary constant 
(between zero and one) that partitions the trust re- 
gion into two sections, one for work with the con- 
straints, the other, for work with the objective. 

This algorithm for computing the trial step is a 
special case of the MAESTRO class with the dis- 
tinction that the Gauss-Newton model of the con- 
straints and the quadratic model of the objective or 
the Lagrangian have been replaced by the general, 
first-order models that satisfy the consistency condi- 
tions (3) and (4). 

For the purposes of global convergence analysis, 
the £2 penalty function 

V(x;p) = f(x) +p || h(x) || 2 


If pred c > hpred c then 
P+ = Pc- 

Else 


P+ — Pc + A 

1 _ 2[al (x c + s c ) — f (x c )] 

where Pc = hprel ■ 

End if 


Set p c — p_|_ . 


Thus, the penalty parameter is increased just suf- 
ficiently to maintain positive predicted reduction. 
Given that the algorithm does not terminate at a par- 
ticular iterate, the penalty parameter can be shown to 
be bounded above. Since, when the penalty parame- 
ter has to be increased, it is increased just enough to 
keep the predicted reduction positive, it is expected 
to grow more slowly than in methods where it is in- 
creased simply by a predetermined factor. It should 
be emphasized that the penalty parameter is used 
only to evaluate the trial step and not in computing 
the step. 


Evaluating the trial step and updating 
the radius 


Once the penalty parameter is updated, the trial 
step is evaluated. It is accepted if it produces an ac- 
tual decrease in the merit function, and it is rejected 
otherwise: 


x + 


x c + s c if V(x c + s c ; p c ) < V(x c ; p c ) , . 
x c otherwise. ' 


suffices as a merit function. The penalty parame- 
ter p balances the progress toward optimality with 
the progress toward feasibility. Global convergence 
analysis is also applicable when the augmented La- 
grangian is used as the merit function. The latter 
would be used for computational efficiency. 

Updating the penalty parameters 

Before the trial step is evaluated, the penalty pa- 
rameter is updated. The parameter p is a positive, 
scale dependent number. The updating scheme is rig- 
orous and it extends the scheme introduced in [30]. 
The penalty parameter is occasionally increased to 
ensure that the predicted reduction in the merit func- 
tion is positive at each iteration. In particular, let us 
set p c = 1 at the beginning and choose [3 £ (0,1). 
Let s c = si + s 2 be the trial step computed during 
some iteration, and let p c be the current value of the 
penalty parameter. Then we update p c as follows: 

Compute hpred c = || h{x c ) || 2 — || a(?(* c + s c ) || 2 

Compute the predicted reduction 

predc = [f(x c ) ~ a{ (*c + s c )] + p c hpred c . 


To adjust the trust-region radius, one considers 
the ratio of the actual reduction defined as 

ared c = V(x c ;p c ) ~ V(x c + s c ; p c ), 

to the predicted reduction pred c : 

ared c 

predc 

When the ratio is close to zero, the predictive be- 
havior of the model is unsatisfactory. The agreement 
between the actual and predicted reduction need not 
be large. For instance, typically, r < 10 -5 is usually 
considered to signify a unsuccessful step. A larger 
ratio indicates satisfactory behavior. The radius is 
updated as follows: positive constants r\ < r 2 < 1 
and ci < 1, c 2 > 1 are chosen to regulate the con- 
traction and expansion of the trust region. Then 

( max{ci || s c || , A*} if r < rq 

min{c 2 || A c || , A*} if r > r 2 (6) 

|| s c || otherwise, 

where A* is a lower bound on the trust-region radius 
and A* is an upper bound. Details on typical values 
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of the parameters for trust-region algorithms can be 
found in [31]. 

One should note that although for the purposes 
of engineering analysis a large r (greater than one) 
indicates a poor match between the higher and lower 
fidelity models, for the purposes of optimization, a 
large r indicates excellent predictive capabilities of 
the model, and the radius A c is increased. 

Details of computing the trial step 

Since the underlying algorithm belongs to the 
MAESTRO class, the MAESTRO-based AMF will 
converge to a critical point of the high-fidelity prob- 
lem under the assumptions that lead to convergence 
of the underlying class. This means Ending the sub- 
steps si and S 2 that will satisfy the sufficient de- 
crease conditions necessary for establishing conver- 
gence. The following procedure will suffice. (We will 
use Vj and pj as local variables in both procedures.) 

Compute the substep si on the constraints in 
an iterative procedure of minimizing Gauss-Newton 
models of within the trust region of that subprob- 
lem, using || a* || as an internal merit function: 

Given z 0 = x c £ ffl 1 , A c > 0, r £ (0, 1), 

and a > 1, set y 0 = zo, So = r A c , i> 0 = 0. 

For j = 0,1,..., while || Vj || < aA*, 

and at least until Vj ^ 0 do { 

Construct 

IjiVj +P) = || 4(Vj) +VaJ(yj) T p f. 

Find an approximate solution pj to 
minimize qj ( yj + p) 
subject to || p || < Sj 

II Vj+P II < A c 

that satisfies FCD for || || from yj . 

Compare the actual and predicted decrease 

a c{yj) f - || a c{yj+Pj) f 

II «c(%) II 2 - QjiVj+Pj) 
Evaluate pj and update yj according to (5) 
and Sj according to (7). 

Set v j+1 = Vj + (y j+ 1 - yj). 

} 

Set si = Vj . 

Thus, the subproblem for computing si is an uncon- 
strained trust-region subproblem. By results of the 
unconstrained AMF [13], the substep si satisfies the 
FCD condition for that subproblem. 

Once si is computed, z i is set to zo + si, and S 2 
can be computed via the following procedure: 

Given z\ 6 5i", A c > 0, r £ (0, 1), and a > 1, 



v 0 = 0. 

v/ATq 


i) T p+\p T B jP , 


set y 0 = zi, S 0 = Tj/\ 2 C - ||si|| 2 , v 0 = 0. 

For j =0,1,.. ., while || vj || < a^/ A 2 C - ||si|| 2 
and at least until Vj ^ 0 do { 

Construct 

9j {yj + p) = 4 ( yj ) +Va f c ( yj ) T p + \p T B jP 
where Bj approximates the second order 
information for a{ at yj . 

Find an approximate solution pj to 
minimize q j (yj + p) 

subject to Vc^(zo ) T p = 0 

Hell < S j 

II %' + ell < A c 
that satisfies FCD for a{ from yj . 

Compare the actual and predicted decrease 
in aB. 

r = 4(yj) ~ 4 ( Vj +Pj) 

4(yj) - q S j (yj +Pj) 

Evaluate pj and update yj according to (5) 
and Sj according to (7). 


j+i = v j + (%+ 1 - yj)- 


Set s 2 = Vj. 

Set s c = si T s 2 . 

To prove the satisfaction of FCD for the resulting 
step s c , a more stringent rule for updating yj is re- 
quired than for updating x c in (5). Namely, p > 0, 
independent of the iterates, is chosen and the iterate 
is updated as follows: 


If yj = x c , then y j+i = l Vj 

l Vj 


If Vj ± 


then 

3+ l yj 


if r > pL 
otherwise, 
if r > 0 
otherwise. 


Note that for computing S 2 , the null-space con- 
straints that preserve the predicted decrease in the 
constraints already obtained during the computa- 
tion of si, are eliminated via a change of variables, 
thus converting the second subproblem into a lower- 
dimensional unconstrained trust-region subproblem. 
By results for the unconstrained AMF, so satisfies 
FCD for its subproblem. 

Note, again, that in the “inner” algorithms, the 
models that are minimized are now the quadratic ap- 
proximations of and , not of h and /, respec- 
tively. 

There are a number of ways to solve the subprob- 
lems in the “inner” iterative procedures. The reader 
is referred to [24, 28] for details. 


Statement of the bilevel AMF algorithm 

The algorithm in its entirety can now be stated 
concisely: 
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Given x c E 4f n , A c > 0, and p c > 1, 

Do until convergence: 

Select the models a* and a£ . 

Compute the trial step s c = si + s 2 - 

Update the penalty parameter p c . 

Evaluate s c and update x c and A c . 

End Do 

Multilevel AMF 

In the context of MDO, the bilevel algorithm is 
easily extended to the general, multilevel case. In 
particular, the constraint system is partitioned into 
M blocks, h i(x), . . .,h,M(x), and the total trial step 
is computed as a sum of M + 1 substeps. 

We describe a multilevel algorithm with general 
first-order models. We will denote the iterates x c 
and will omit the subscripts denoting the iteration 
number, for convenience of notation. Instead, the 
subscripts will denote entities concerning a partic- 
ular constraint block, e.g., we let a\, . . . , denote 
the models of the constraint blocks hi (x ) , . . . , /im (%) , 
respectively. Then the algorithm can be stated as fol- 
lows: 

Given x c £'?R n ,6k>0,k=l,...,M+l, p k >0,k = 

1 

Outer Loop: Do until convergence: 

Zo = X c 

Compute the trial step 

Inner Loop: Do k = 1, M 

Select a k such that a k (zk-i) = h k (zk-i) 
and Va^(zk-i) = V/i fe (zfc-i) 

Compute Sk that approximately solves: 
minimize a^(zfe_i + s) 

subject to Vay (zy_i ) T s = 0, j = 1, . . . , k — 1 

|| s || < S k 
Set z k — %k — l T 

End Inner Loop 

Step on the objective function: 

Select a? such that a? (zm) = /(^m) 
and Va f (z M ) = V/(z M ) 

Compute sm + i that approximately solves: 
minimize a? (zm + s) 

subject to Va^ (zj_i) T s = 0, j = 1, . . . , M 
|| -s || < S M +i 

The trial step is: s c = si + . . . + sm+i 

Update the penalty parameters 

Evaluate the step and update 

the trust-region radii 
End Outer Loop 

In summary, at each iteration, for each of the sub- 
problems, a lower-fidelity approximation is selected 


that satisfies the consistency conditions for that par- 
ticular model at the most recently computed point. 
Then unconstrained procedures described in the sec- 
tion on bilevel AMF are applied to models of each 
constraint block, followed by the unconstrained pro- 
cedure on the model of the objective function. The 
procedures are unconstrained because the null-space 
constraints on all of the subproblems are eliminated 
via a change of variables. The individual substeps 
s k ,k = 1, . . M + 1, satisfy FCD for the subprob- 
lems they solve. 

The individual merit functions are modified in an 
obvious manner. The complete merit function of the 
multilevel algorithm has to be used with the appropri- 
ate “staged” penalty parameter updating procedure. 
The reader is referred to [25], for example, for details 
on the total merit function and procedures for up- 
dating the penalty parameters and the trust-region 
radii. 

Preliminary numerical results 

The algorithm has undergone initial testing on 
several simple problems. Table 1 illustrates prelim- 
inary results. Problems denoted by “HSxx” are the 
problems from the Hock and Schittkowski test suite 
for nonlinear optimization [32, 33]. All problems 
are small, algebraic, nonlinear programming prob- 
lems with well-known solutions. 

The entries in the tables are the numbers of func- 
tion evaluations taken by each method to attain a 
known solution of a particular problem. 

The problems were initially solved using the well- 
known commercial optimization package NPSOL [34] 
that improves the global properties of the SQP al- 
gorithm via the use of line searches. The problems 
were then solved using a research implementation 
of MAESTRO without approximation management. 
For NPSOL and MAESTRO, the number of function 
evaluations is reported. “F” denotes failure to con- 
verge in a thousand iterations. 

Finally, the problems were solved with the 
MAESTRO-based AMF approach. Because the 
present demonstrations deal with physical models 
of varying fidelity, this very simple test used func- 
tion evaluations computed to machine precision as 
the “high-fidelity” models, and function evaluations 
with abbreviated precision and added noise as “lower- 
fidelity” models. A variety of precisions were at- 
tempted. On average, the “lower-fidelity” functions 
have from two to four digits of accuracy after the 
decimal point. The number of “high-fidelity” evalu- 
ations is reported with the total number of evalua- 
tions, including the number “lower-fidelity” evalua- 
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tions, given in parentheses. 

Given the initial problem set, the results sug- 
gest that the MAESTRO-based AMF tends not im- 
pact the performance of the underlying algorithm sig- 
nificantly and it does reduce the number of “high- 
fidelity” function evaluations. 

In order to draw definitive conclusions, a consid- 
erable amount of experimentation with physics-based 
models will be required. 

At present, we are continuing testing the method 
on a subset of the MDO Test Suite [35] and we are in 
the process of demonstrating AMF on a more realistic 
problem of aerodynamic wing design. 

Concluding remarks 

In summary, we introduced an approximation 
management framework for solving constrained non- 
linear problems based on an extension of the MAE- 
STRO algorithms for nonlinear programming and 
MDO. 

Global convergence of the AMF to a critical point 
of the original high-fidelity problem is an immediate 
consequence of the convergence results for underlying 
MAESTRO algorithms and the results for the uncon- 
strained AMF. 
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Problem 

Starting Point 


Fn Evaluations 1 



NPSOL 

MAESTRO 
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-1.2 1 

20 

11 

6(14) 


12 10 

11 

11 

5(11) 


-10 0 

30 

13 

6(17) 

HS7 

2 2 

17 

11 

6(11) 


-35 -40 

41 

33 

27(40) 


-15 -6 

59 

17 

10(11) 

HS26 

0 0 0 

49 

21 

15(25) 


5-5 5 

105 

32 
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30 35 40 

70 

21 

17(21) 

HS39 

2 2 2 2 

16 

19 

8(24) 


40 2 4 -5 

320 
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61(133) 


-2-4 6 2 
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19 
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-1 -1 -1 -1 
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8(12) 


30 29 -39 3 

40 

61 

44(83) 

HS60 

2 2 2 

12 

11 

9(14) 


-10 40 9 

124 

21 

11(36) 


100 100 -100 

F 

21 

16(40) 

HS77 

2 2 2 2 2 

21 

11 

6(17) 


10 10 10 10 10 
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20 20 20 20 20 

F 

27 
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